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The EICCD Program Evaluation Process 2 
Abstract 



The primary purpose of the EICCD Program Evaluation Process is to ensure 
the effective and efficient use of resources, and instructional quality and 
curricular relevance. Developed and pilot-tested on thirty-eight vocational- 
technical programs, the model is being utilized by more than 120 technical and 
community colleges in the U.S. and Canada. The process consists of two phases: 
1) an annual review of the program on ten data elements, and 2) an in-depth 
program review conducted once every 3-5 years. Phase II consists of: faculty 
self-study; an internal review team visit and report; surveys conducted of six 
target groups; and, decision making. 
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The EICCD Program Evaluation Process: 
A Primary Data Source for 
Strategic Planning and Decision Making 

No one can deny the impact of the changing and emerging technologies, and 
the shifting of the U.S. economy from a heavy manufacturing foundation to the 
technicial and service industries on our nation's community and technical 
colleges. An influx of non-traditional students across a wide spectrum of 
entry-level skills and knowledge, and severe fiscal restraints, have further 
stretched the ability of community and technical colleges to provide quality 
educational opportunities for all students. Institutions must be able to 
respond programmatically to these challenges; however, what information do 
decision makers need in order to plan strategically and to act tactically? The 
Eastern Iowa Community College District (EICCD) has developed and implemented a 
program evaluation process which provides timely and accurate data to both 
faculty and administrators regarding the health and viability of its programs, 
and program recommendations are formulated and validated. 

The pilot test of the EICCD Program Evaluation Process on 36 vocational- 
technical programs was completed over a two and a half year period. Its 
primary purpose is to ensure the effective and efficient use of resources, and 
instructional quality and curricular relevance. Developed principally for 
vocational and technical programs, the information garnered from the 
evaluations is used to assist in program and cost accountability efforts, and 
to provide direction to program/curriculum development, modification or 
termination. Additionally, the implementation of the EICCD Program Evaluation 
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Process has provided opportunities to compare business and industrial standard^ 
with occupational program standards; to involve and utilize the expertise of 
representatives from related businesses and industries in the program and the 
college; to inform former students, employers, and the business community of 
the institution's commitment to self-improvement and excellence; and, to 
identify needed changes and modifications in its physical facilities and 
instructional equipment. 

The purpose of this article is to briefly describe the development of the 
process, its major components, and its use by the EICCD and other technical and 
community colleges. 
Definition of Program Evaluation 

There are many different definitions for evaluation. Some definitions 
regard evaluation as roughly synonymous with educational measurement (Thorndike 
& Hagen, 1969; Ebel, 1967). Other definitions view evaluation as a process of 
identifying and collecting information to assist decision-makers in choosing 
among available decision alternatives (Stake, 1967; Provus, 1969; Stuff lebeam, 
et al, 1971). Very simply, evaluation is the determination of the worth of a 
thing. "It includes obtaining information for use in judging the worth of a 
program, product, procedure, or objective, or the potential utility of 
alternative approaches designed to attain specified objectives." (Worthen, page 
19). 

Wendle and Neigher (1978) distinguished three models for the use of 
evaluation information: accountability, advocacy, and program improvement. 
For the EICCD, the major goal of program evaluation is to improve the quality 
of education; secondarily, it is accountability. 
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The Purpose of Program Evaluation 

The major goal of program evaluation for the EICCD is program improvement. 
It is a means of ensuring program quality and relevance, and the effective and 
efficient use of resources. Program evaluation is . a cooperative process which 
utilizes the knowledge and expertise of instructors, administrators, current 
and former students, employers, and advisory committee members. Program 
evaluation is worthwhile in itself for it initiates a process of self- 
examination among administrators and faculty, and assists in the formulation 
and clarification of program goals and objectives. / 

The EICCD Program Evaluation Process provides for the flow of information 
and recommendations from the faculty, through the administrative ranks, all the 
way to the Chancellor of the District. Not only is the information used by 
decision-makers regarding the maintenance, modification or termination of an 
existing program, but it also provides valuable insights into areas of new 
program development. 

The Eastern Iowa Community College District 

The EICCD is a multi-college District serving eight counties in east- 
central Iowa, and . is comprised of three colleges: Scott Community College, 
Clinton Community College, and Muscatine Community College. Each of the three 
colleges is a comprehensive community college with Arts and Science college 
transfer programs-, vocational-technical programs, and community /continuing 
education offerings. The credit offerings include 36 vocational-technical 
programs, 11 career option/college parallel programs and more than 70 Arts and 
Sciences concentrations. 

The District Office of Academic Affairs and Planning coordinates the 
program development, revision, and evaluation efforts of all three colleges, 
and includes the functions of institutional research. 
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Development of the EICCD Program Evaluation Process 

The successful development and implementation of a comprehensive program 
evaluation process for the vocational-technical offerings of the EICCD was 
contingent upon the involvement of and ownership by faculty in the process, and 
administrative support and commitment. 

In 1985, the EICCD established a Program Improvement Steering Committee 
with the purpose of designing a program evaluation process which would provide 
program specific information to all levels of institutional decision-makers. 
This Committee was composed of faculty, college program administrators, Student 
Development personnel, and District Academic Affairs administrators The 
purpose of the Committee was to develop a practical program evaluation process 
which would include the collection and analysis of pertinent program data; 
provide recommendations regarding a program's curriculum, facilities and 
equipment, staff developmei t activities for faculty, and college/student 
services; and, require minimal time on the part of faculty. 

The District Office of Academic Affairs and Planning conducted a review of 

the literature and related research regarding program evaluation processes, 

looking specifically at those utilized for vocational and technical programs, 

and by community and technical colleges. Over 40 different models were 

examined in detail, commonalities and strengths noted, summary reports written, 

« 

and a review conducted by the Program Improvement Steering Committee. 
Visitations to three colleges with operational program evaluation processes 
were conducted by various members of the Committee. 
Obstacles to Overcome in the Development of the Process 

The EICCD needed to develop a process which would provide faculty the 
opportunity to respond to data regarding their program, to identify the 
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strengths and weaknesses of their program^ and to make recommendations. In the 
EICCD, the teachers 1 Master Contract stipulates that program evaluation is a 
responsibility of a full-time teacher; teachers are not paid extra for 
completing tasks related to the evaluation of their program. Therefore, the 
EICCD needed to develop a process which would not require an inordinate amount 
of faculty time. 

Another obstacle facing the EICCD was a faculty mistrust of statistical 
data regarding their program' s enrollment and cost. Objectivity and 
consistency in reporting was a primary concern. The Committee recommended that 
these responsibilities be assigned to the District institutional research staff 
to ensure objectivity, accuracy, and consistency in reporting data across all 
programs. 

In order to limit the amount of time required of faculty in the 
implementation of the process, all surveys conducted as a part of the process 
would be mailed, receipted, tabulated, analyzed, and a summary report written 
by staff in the District Office of Academic Affairs and Planning. This would 
also guarantee accuracy and objectivity in the collection and reporting of 
survey data. 

The EICCD Program Evaluation Process 

The EICCD Program Evaluation Process has been pilot-tested and evaluated 
on the thirty-six vocational-technical programs of the District. It has proven 
to be comprehensive in nature, low in cost, and require minimal additional time 
on the part of faculty. Budgetary decisions regarding facility changes, 
equipment acquisition, and staff development of faculty are based upon the 
results of these evaluations. 
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The EICCD Program Evaluation Process consists of two phases: Phase I, an 
Annual Review of the program; and, Phase II, an in-depth evaluation conducted 
on a three-year cycle. 
Phase I: Annual Review 

The purpose of the Annual Review is to provide a "snapshot view 11 of the 
health of a program. This two-page report on each program is a quick means of 
determining the well-being or health of a program. The ten data elements of 
the Annual Review are compared year-to-year over a three-year period* These 
data elements are: 

1. Unduplicated enrollment/headcount 

2. Contact hours generated by the program 

3. FTEE generated by the program 

4. Program graduation/completion rates 

5. Program leavers and withdrawal rates 

6* Program majors 1 intent for enrolling in the program 

7. Program costs (per student, contact hour, and FTEE) 

8. Average class size 

9. Success of program leavers and graduates (includes job placement and 
employment status, as well as college transfer information) 

10. advisory Committee meetings 1 highlights and Department meetings 1 
highlightG 

The collection and reporting of the annual review data is completed by the 
District Office of Academic Affairs and Planning. Printed reports are provided 
to college administrators annually, and to faculty when uhey are conducting the 
Self-Study of their program. 
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P hase II; In-Step Program Review 

All vocational-technical programs are evaluated on a three-year cycle. 
The sequence of programs in the three-year cycle is determined by the District 
Instructional Council, but a program's placement in the cycle may be 
accelerated if the Annual Review raises questions or concerns regarding its 
continued viability. 

The five major components of Phase II are: 

1. Surveys conducted of six target groups. 

2. A local labor market assessment in the specific program/occupational 
area. 

3. Faculty Self-Study of the program. 

4. The Internal Review Team visit and report. 

5. Faculty Response. 

1. Surveys . Surveys are conducted of the six groups prior to the Faculty 
Self-Study: 1) current students enrolled in the program; 2) graduates of the 
program from the past three years; 3) college administrators; 4) part-time 
faculty in the program; 5) program Advisory Committee members; and, 
6) employers of the program's graduates. 

The target groups are asked to provide information regarding the quality 
of instruction, program and course objectives, instructional methodologies and 
materials, equipment, facilities, advising, job placement and college, transfer 
services, individualized instruction, learning resources , other college 
services, and Tuture occupational outlook. 

These surveys are standardized for all vocational- technical programs of 
the EICCD, and are designed for tabulation and analysis using the SPSS . 

The distribution and receipting of the surveys; data entry, analysis, and 

V 

ERIC J1 



The EICCD Program Evaluation Process 10 

interpretation; and the writing and printing of the results are completed by 
the EICCD Office of Academic Affairs and Planning. Reports are provided to Lhe 
faculty and Internal Review Team members for interpretation, the identification 
of program strengths and weaknesses, and the formulation of recommendations. 
Thus, faculty are put into a position of interpreting and recommending changes 
in their program, as opposed to data collection and analysis. 

2. Local Labor Market Assessment . A local labor market assessment is 
conducted in each program's occupational field. Combined with regional and 
national data, the assessment provides infoxm&tion regarding employer need for 
the program's graduates as well as future needs in the occupational field. The 
local labor market assessment is unique for each program/occupational field. 

The development of each program's labor market assessment, and the data 
collection and analysis is coordinated by the District Office of Academic 
Affairs and Planning. An Ad Hoc Assessment Task Force is formed for each 
occupatioral/program area, composed of faculty, employers, and an Assessment. 
Specialist ♦ Since the information from the assessments is used for 
programmatic as well as budgetary decision making, it is important that faculty 
ars involved in the identification of questions specific to the occupational 
field and the determination of businesses and industries who will receive the 
survey. 

The assessment consists of two surveys. The initial survey is sent to all 
known or prospective employers of technicians or specialists in the 
occupational field. This survey obtains information regarding: 

1. The demographics of the respondent: location, type of business, size, 
total number of employees, and the number of employees in specific 
categories. 
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2. Entry-level employees: minimum education evel required and 
preferred, minimum work experience required and preferred, type of 
compensation, and expected annual compensation. 

3. Current employees: type of equipment used, continuing education 
needs, and preferred methods for meeting continuing education needs. 

4. Employment needs and projections: immediate number of job openings, 

J 

and projected job openings of the next three years. 

5. Local and national trends in the occupational field. 

The second survey is designed to determine the competencies required of 
entry-level employees in the occupational field. Data are collected from 
employers who indicated on the first survey that they intend to hire employees 
in the specific occupational field. The Ad Hoc Assessment Task Force reviews 
the results of the initial survey giving special attention to the 
characteristics of the respondents, the requirements for employees with 
generalized and specialized skills and knowledge, and the recent and projected 
changes in the occupational field. The Ad Hoc Assessment Task Force also 
examines related research, occupational task lists such as DACUM charts, and 
competency-based curricular materials in the program. The second survey 
consists of the list of entry-level employee competencies compiled by the Task 
Force; the respondents are asked to rate each competency on three variables: 

1. Importance of the competency. 

2. Frequency of use on-the-job. 

3. Immediacy of use. 

This survey also provides space for the respondents to write in additional 
competencies they require new hires to have. 

All labor market assessment surveys are designed for tabulation and 

O 
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analysis using the SPSS . Summary Reports are provided to the faculty and 
Internal Review Team members for interpretation and the formulation of 
recommendations • 

3. Faculty Self-Study , The faculty in each program collectively complete 
the 22 questions of the Self -Study Report. Generally, the Department 
Chairperson coordinates the efforts of the faculty in completing the Self- 
Study. Their responses highlight: 

- the history, philosophy, goals, and objectives of the program; 

- how the program reflects "the state of the art"; 

- teaching methods utilized; 

- type of equipment and facilities currently used; 

- advising of students and job placement of graduates: 

- qualifications, certification and staff development activities of full- 
time and part-time faculty; 

- the strengths and weaknesses of the program; 

- factors impacting the program; 

- additional equipment and/or facilities needed to maintain and/or 
enhance the program; 

- and, recommended curriculum/program changes. 

Before completing the Self-Study, the faculty review the results of the 
local labor market assessment and other national and regional data collected 
regarding the occupational field, as well as the results of the surveys 
conducted as a part of the program evaluation process. Use of these materials 
assist the faculty in the identification of their program strengths and 
weaknesses, and the formulation of their recommendations. The Course 
Development Models (course syllabi) for all courses specific to the program 
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major are inserted into the Faculty Self-Study. The Report is then forwarded 
lo and reviewed fay an Internal Review Team, 

4. The Internal Review Team . An Internal Review Team is designated for 
each specific program. Minimally, the team is composed of four individuals 
internal to the District but external to the faculty of the program; thus, in 
many ways, i' serves the purpose of external evaluation, but at no additional 
cost to the District. An external consultant may be involved in the evaluation 
if the team deems it necessary. 

The Internal Review Team generally consists of one District administrator, 
one college administrator, and at least two faculty members (one Arts and 
Science and one Vocational-Technical faculty member) . At least two of the 
teams members are very knowledgeable in the program/occupational field. The 
other two have expertise in curriculum development or other knowledge and 
skills deemed appropriate by the District Instructional Council. Team members 
are selected by the District Instructional Council. 

The Internal Review Team studies all documents related to the evaluation 
of the program. After the study, they conduct an on-site visit to the program 
in which they: discuss the program and the content of the Faculty Self-Study 
with the faculty members and administrators; tour the laboratories; examine the 
course development models and other instructional materials, equipment, and 
supplies; interview current students; and meet with Advisory Committee members 
regarding their role and e perceptions of the program. 

The Team then completes a written report of their findings and 
recommendations. This report consists of the answers to 36 questions 
paralleling the questions to which the faculty have responded. 

The Internal Review Team Report discusses the following aspects of the 

erIc is 
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Program: the consistency of the program goals and objectives with validated 
entry- level occupational competencies; the quality of the curriculum and the 
course development models; the degree to which prerequisite knowledge and 
skills are defined for students; accommodation for part-time, non-traditional, 
and special needs students; success of program graduates and leavers; the locdtl 
and regional employment prospects in the occupational/program field; the 
relevancy, quality, and availability of instructional equipment and materials; 
linkages with businesses and industries; faulty staff development activities 
provided to maintain their knowledge and skills in the occupational field; the 
role of the Advisory Committee; and the adequacy of the program 1 s budget and 
the cost effectiveness of the program. 

The Internal Review Team Report also includes a 15-item rating scale for 
the physical facilities, and if appropriate, can be modified for specific 
programs. 

The Team Report concludes with the identification of the program 1 s major 
strengths, the major factors influencing the continued success of the program, 
and recommendations for improvement. Their completed report is sent to the 
District Office of Academic Affairs and Planning. 

5. Faculty Response . Upon receipt of the Internal Review Team Report, 
the District Office of Academic Affairs and Planning forwards a copy to the 
chairperson of the Faculty Self-Study Committee, as well as to the college Dean 
of Instruction. If the faculty disagree with statements made in the Internal 
Review Team Report, they have the opportunity to respond in writing within a 
specified period of time. This response is sent to the District Office of 
Academic Affairs and Planning for consideration in the decision-making process. 
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The Decision Making Process 

Copies of the Faculty Self-Study Report, the Internal Review Team Report, 
and the Faculty Response (if any) are disseminated by the District Office of 
Academic Affairs and Planning to the college Presidents and the Deans of 
Instruction. Their reactions to the recommendations formulated by the faculty 
and the Internal Review Team are forwarded to the District Office of Academic 
Affairs and Planning. 

All or the documents compiled as a part of the program evaluation process 
are made available to the administration of the colleges and the District. 
Bpcau^B of the volume of information that is generated, staff from the District 
Office of Academic Affairs and Planning review all reports, and compile a 
sumuary of each program 1 s evaluation. The summary reports highlight the 
strengths and weaknesses of the program as cited by each group, as well as the 
recommendations. The summary reports are sent to all members of the 
Instructional and Administrative Councils, as well as to the Chancellor of the 
District. 

A joint meeting of the District 1 s Administrative and Instructional 
Councils is chaired by the Chancellor. All individuals present have copies of 
the summary reports, as well as access to the complete reports. These meetings 
are usually one day in length. The evaluations of all programs are discussed, 
and decisions are made regarding new program direction, the acquisition of new 
equipment , facilities requirements , and staff development activities for 
faculty. 
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Use of the Evaluation Results 

The information gained through the evaluations have provided direction* to 
the development of the EICCD T s vocational-technical programs to a competency- 
base. All programs have been modified based on the information obtained 
through the evaluations; a few have been terminated. Eight new programs have 
been developed: Sales and Marketing, Graphic and Printing Communications ? 
Hazardous Materials Technology, Chemical Laboratory Technician, Travel 
Specialist, Management and Supervision, Microcomputer Software Applications, 
and Automated Manufacturing. 

The results of the evaluations are used in determining the relationship of 
the program's student outcomes to employers' needs; the subsequent program 
revisions are vital to maintaining quality and relevant programs. These 
revisions include: 

- The incorporation of computer-aided design into the Drafting program. 

- The addition of computer-aided design instruction, as well as 
computerized inventory control in the Interior Design program. 

- The incorporation of information regarding computerized controls and 
systems in residential and commercial heating and air conditioning in 
the Heating, Ventilation and Air Conditioning program. 

- The revision of the Automotive Technology program into a ladder 
structure which awards a certificate after one semester, a diploma 
after one year, and an A.A.S. after completion of the two-year program. 
The evaluation also indicated that the Automotive Technology program 
should become ASE certified; through the subsequent curriculum 
development, this program is now ASE certified in all eight areas. 

ERIC J S 



The EICCD Program Evaluation Process 17 

- The addition of instruction in satellite communications and 
microcomputer repair in the Electronics program. 

- The development of computer-assisted instructional units and the 
addition of units related to geriatric care in tha Nursing curriculum. 

The EICCD has designated that one-third of its annual equipment 
acquisition budget be tied to the equipment approved for purchase through the 
program evaluation process. Funds for^ faculty development activities are also 
allocated according to the needs cited in the evaluations. 

The evaluations may also highlight deficiencies in other areas of the 
college, such as job placement services, advising and counseling, financial 
aid, child care, etc. 

The administration of the EICCD has learned that the evaluation process 
brings people together. Faculty, administrators, employers, and students are 
all involved in the process. It fosters cohesion across the institution by 
involving faculty external to the program being evaluated. Decision making has 
become a participatory process; data regarding programs are open to scrutiny 
and review at all levels of the college community. 

The EICCD Program Evaluation Process has fostered both institutional unity 
within the college and with the broader community. The process has 
communicated to the community the District's commitment to excellence, and its 
responsiveness to local needs. Secondary benefits that the EICCD has 
experienced includ3 the donation of equipment and supplies to various programs 
from local businesses and industries; and, a 22% increase in the number of job 
listings from employers for program graduates as a result of contact made 
through this process. 

The EICCD Program Evaluation Process can be easily modified and adapted to 
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the needs of other educational institutions. Over 120 technical and community 
colleges in the United States and Canada have utilized the EICCD Program 
Evaluation Process in their efforts to obtain relevant information which 
fosters quality decision making and strategic planning. 
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